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ABSTRACT 

This  paper  presents  a  super-resolution  image 
reconstruction  from  a  sequence  of  aliased  imagery. 
The  sub-pixel  shifts  (displacement)  among  the 
images  are  unknown  due  to  uncontrolled  natural  jitter 
of  the  imager.  A  correlation  method  is  utilized  to 
estimate  sub-pixel  shifts  between  each  low  resolution 
aliased  image  with  respect  to  a  reference  image.  An 
error-energy  reduction  algorithm  is  derived  to 
reconstruct  the  high-resolution  alias-free  output 
image  by  imposing  the  spatial  domain  constraint  and 
the  spatial  frequency  domain  constraint  in  an  iterative 
fashion  to  form  the  desired  alias-free  (super-resolved) 
image.  The  results  of  testing  the  proposed  algorithm 
on  the  simulated  low  resolution  FLIR  (Forward- 
Looking  Infrared)  images  and  real  world  FLIR 
images  are  provided. 

1.  INTRODUCTION 

Many  low-cost  sensors  spatially  or  electronically 
undersample  an  image.  This  results  in  aliased 
imagery  in  which  the  high  frequency  components  are 
folded  into  the  low  frequency  components  in  the 
image.  Consequently,  subtle/detail  information  (high 
frequency  components)  are  lost  in  these  images.  An 
image/signal  processing  method,  called  super¬ 
resolution  image  reconstruction,  can  increase  image 
resolution  without  changing  the  design  of  the  optics 
and  the  detectors.  In  other  words,  super-resolution 
image  reconstruction  can  produce  high-resolution 
images  by  using  the  existing  low-cost  imaging 
devices  from  a  sequence  (or  a  few  snapshots)  of  low 
resolution  images.  The  emphasis  of  the  super¬ 
resolution  image  reconstruction  algorithm  is  to  de¬ 
alias  the  undersampled  images  to  obtain  an  alias-free 
or,  as  identified  in  the  literature,  a  super-resolved 
image.  This  process  also  increases  the  image 
bandwidth  which  is  limited  by  the  imager  post  filter. 

When  undersampled  images  have  sub-pixel 
shifts  between  successive  frames,  they  represent 


different  information  from  the  same  scene.  Therefore, 
the  information  that  is  contained  in  undersampled 
image  sequence  can  be  combined  to  obtain  an  alias- 
free  (high-resolution)  image.  Super-resolution  image 
reconstruction  from  multiple  snapshots  provides  far 
more  detail  information  than  any  interpolated  image 
from  a  single  snapshot. 

There  are  three  major  steps  in  super-resolution 
image  reconstruction  methods  (Park,  2003;  Schuler 
and  Scribner,  2000;  Borman  and  Stevenson,  1999). 
They  are: 

1)  Acquiring  a  sequence  of  images  from  the 
same  scene  with  sub-pixel  shifts  (fraction  pixel 
displacements)  among  the  images. 

2)  Estimating  the  sub-pixel  (fraction  pixel) 
shift  or  displacements  among  the  images. 

3)  Reconstructing  the  high-resolution  image. 

In  the  first  step,  there  are  two  types  of  methods 
to  acquire  low  resolution  images  with  sub-pixel  shifts 
among  them.  One  method  is  to  have  a  controlled 
pixel  displacement  (Zalevsky,  2004).  In  this  method, 
a  special  sensor  or  scanner  (hardware)  is  designed  to 
capture  multiple  images  in  a  known  pattern,  where 
each  image  is  captured  by  displacing  the  sensor  in  a 
known  distance  that  is  not  a  multiple  of  a  pixel,  but 
rather  is  a  multiple  of  a  pixel  plus  a  known  fraction  of 
a  pixel.  Another  method  is  to  have  a  non-controlled 
pixel  displacement,  e.g.  natural  jitter.  This  paper 
considers  the  natural  jitter  which  is  more  cost 
effective  and  practical.  For  example,  in  many 
applications,  an  imager  is  carried  by  a  moving 
platform.  In  a  rescue  mission,  the  camera  may  be 
carried  by  a  helicopter,  or  a  moving  vehicle.  In  a 
military  reconnaissance  situation,  the  camera  may  be 
carried  by  a  person,  an  unmanned  ground  vehicle 
(UGV),  or  an  unmanned  aerial  vehicle  (UAV). 

The  second  step  is  to  estimate  the  sub-pixel  shift 
or  fraction  pixel  displacements.  There  are  many 
methods  that  are  addressed  in  the  literature.  Frame- 
to-frame  motion  detection  based  on  gradient  decent 
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methods  are  most  used  (Horn  and  Schunk,  1981; 
Irani  and  Peleg,  1991).  Most  of  these  methods  need 
to  calculate  the  matrix  inversion  or  use  an  iterative 
method  to  calculate  the  motion  vectors.  In  this  paper, 
we  utilize  a  correlation  method  to  explore  the 
translational  differences  (shifts  in  x  and  y  domains) 

of  Fourier  transforms  of  two  images  to  estimate  sub¬ 
pixel  shifts  between  two  low  resolution  aliased 
images. 

The  third  step  in  super-resolution  image 
reconstruction  is  to  reconstruct  the  high-resolution 
image.  Many  methods  have  been  proposed  to  solve 
the  problem.  There  can  be  divided  into  two 
categories:  directive  non-uniform  interpolation  and 
non-directive  inverse  processing.  Directive 
interpolation  methods  are  described  using  spatial 
interpolation  at  the  inter-pixel  position  (Peleg  et  al., 
1987),  using  warping  procedure  (Chiang  and  Boult, 
2000),  and  using  weighted  interpolation  (Alam  et  al., 
2000).  In  non-directive  inverse  processing  methods, 
an  observation  model  is  formulated  to  relate  the 
original  high-resolution  image  to  the  observed  low 
resolution  images.  The  solution  of  high-resolution 
image  is  often  obtained  by  inverting  a  matrix  or  an 
iterative  procedure.  The  observation  model  can  be 
formulated  in  either  the  spatial  or  frequency  domain. 
The  corresponding  inverse  methods  are  implemented 
in  both  domains.  These  methods  are  called 
regularized  super-resolution  approaches  (Tom  and 
Katsaggelos,  1995). 

This  paper  outlines  an  iterative  error-energy 
reduction  method  to  reconstruct  the  high-resolution 
image.  The  algorithm  utilizes  a  correlation  method  to 
estimate  sub-pixel  shifts  among  the  members  of  the 
acquired  image  sequence.  We  begin  with  a  signal 
model  to  relate  the  aliased  images  via  gross  shifts  and 
sub-pixel  shifts  (Section  2.). 

The  knowledge  of  the  location  of  each  low 
resolution  image  in  the  upsampled  grid  (processing 
array)  is  the  basis  of  an  iterative  error-energy 
reduction  algorithm  to  reconstruct  the  high-resolution 
(alias-free)  image  (Section  3.).  For  this  purpose, 
spatial  domain  constraints  (knowledge  of  samples  of 
aliased  images  and  their  locations  on  the  upsampled 
array)  and  spatial  frequency  domain  constraint 
(bandwidth)  are  imposed  in  an  iterative  fashion  to 
form  the  desired  alias-free  (super-resolved)  image. 

In  Section  4,  we  provide  experimental  results 
using  simulated  low  resolution  FLIR  images  and  real 
world  FLIR  images  where  the  sub-pixel  shifts  among 
frames  are  caused  by  natural  jittering  of  the  imager. 
Conclusion  is  provided  in  Section  5. 


2.  SHIFT  ESTIMATION 

An  overview  of  the  super-resolution  image 
reconstruction  algorithm  is  illustrated  in  Fig.  1.  A 
sequence  of  original  input  low  resolution  images  is 
passed  into  the  gross  shift  estimation  algorithm  to 
estimate  the  overall  shift  of  each  frame  with  respect 
to  a  reference  frame.  The  undersampled  low 
resolution  images  are  captured  either  by  natural  jitter 
or  some  kind  of  controlled  motion  of  the  camera.  The 
images  of  successive  frames  contain  not  only  the  sub¬ 
pixel  shifts  but  also  the  integer  pixel  shifts.  Before 
estimating  the  sub-pixel  shifts,  the  gross  shifts  among 
frames  are  compensated.  After  the  gross-shifts  are 
estimated,  the  input  images  are  realigned.  Then  the 
sub-pixel  shifts  are  estimated  for  each  frame  with 
respect  to  a  reference  frame.  The  error-energy 
reduction  algorithm  is  applied  to  the  low  resolution 
input  images  with  the  estimated  sub-pixel  shifts 
among  images  to  obtain  the  high-resolution  (alias- 
free)  output.  The  output  is  either  a  single  high- 
resolution  image  that  is  generated  from  a  sequence  of 
low  resolution  images  or  a  sequence  of  high- 
resolution  images  such  as  in  a  video  sequence  that  are 
generated  from  multiple  sequences  of  low  resolution 
images. 

The  shifts  between  two  images  are  estimated 
using  a  correlation  method  to  explore  the 
translational  differences  (shifts  in  X  and  y 

domains)  of  Fourier  transforms  of  two  images.  The 
operation  of  the  sub-pixel  shift  estimation  is  identical 
to  the  gross  shift  estimation.  However,  in  order  to 
achieve  sub-pixel  accuracy,  the  images  are 
upsampled  first.  The  upsampling  is  obtained  by  a 
Fourier  windowing  method  (Young,  2004).  In  this 
Fourier  windowing  method,  the  zero-padding  with  a 
window  is  applied  to  the  Fourier  transform  of  the 
input  image  to  have  an  alias-free  upsampled  image. 
Then  the  correlation  method  is  applied  to  two 
upsampled  Fourier  transforms  of  two  input  images  to 
estimate  the  shifts  in  sub-pixel  accuracy. 

An  example  of  sub-pixel  shift  estimation  from 
16  frames  is  shown  in  Fig.  2.  The  sub-pixel  shift  of 
each  frame  with  respect  to  a  reference  frame  is 
illustrated  in  the  Fig.  2.  From  this  figure,  we  can  see 
that  the  sub-pixel  shifts  among  frames  are  randomly 
distributed  in  both  X  and  y  domains.  The  sub-pixel 

motion  among  frames  is  not  controlled  in  a  fixed 
pattern  in  this  example. 
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3.  ERROR-ENERGY  REDUCTION 
RECONSTRUCTION  ALGORITHM 

In  the  error-energy  reduction  algorithm,  the  high- 
resolution  image  values  are  reconstructed  by 
removing  aliasing  from  low  resolution  images.  The 
algorithm  is  based  on  the  concept  that  the  error- 
energy  is  reduced  by  imposing  both  spatial  and 
spatial  frequency  domain  constraints:  samples  from 
low-resolution  images;  bandwidth  of  the  high- 
resolution  alias-free  output  image.  The  error-energy 
reduction  algorithm  has  been  utilized  in  other  signal 
processing  applications.  Some  of  the  well  known 
examples  included  the  works  by  Papoulis  (Papoulis, 
1975),  Gerchberg  (Gerchberg,  1974),  and  Stark  et.  al. 
(Stark  and  Oskoui,  1989).  Papoulis  utilized  the 
available  information  both  in  spatial  and  frequency 
domains  to  extrapolate  the  distribution  of  auto¬ 
correlation  for  spectral  estimation.  In  Gerchberg’ s 
work,  in  order  to  achieve  resolution  beyond  the 
diffraction  limit,  the  spectrum  of  the  reconstructed 
object  is  enhanced  (expanded)  by  reducing  the  energy 
of  the  error  spectrum  from  one  snap-shot  (not  a 
sequence).  In  the  work  by  Stark  and  his  associates, 
they  described  the  projection  onto  convex  sets 
(POCS)  approach  to  combine  the  spatial  observation 
model  from  a  controlled  rotation  or  scan  and  spatial 
domain  constraints  to  achieve  a  high-resolution 
image. 

The  main  feature  of  this  proposed  error-energy 
reduction  algorithm  is  that  we  treat  the  spatial 
samples  from  low  resolution  images  as  a  set  of 
constraints  to  populate  an  over-sampled  processing 
array;  the  processing  array  is  sampled  above  the 
desired  output  bandwidth.  The  estimated  sub-pixel 
locations  of  these  samples  and  their  values  constitute 
a  spatial  domain  constraint.  Furthermore,  the 
bandwidth  of  the  alias-free  image  (or  the  sensor 
imposed  bandwidth)  is  the  criterion  used  as  a  spatial 
frequency  domain  constraint  on  the  over- sampled 
processing  array.  One  may  also  incorporate  other 
spatial  or  spatial  frequency  domain  constraints  that 
have  been  used  by  others;  e.g.,  positivity  of  the  image 
in  the  spatial  domain.  (This  constraint  is  similar  to 
the  amplitude  constraint  that  was  used  by  Stark  et. 
al.) 

3.1  Image  Acquisition  Model 

Fig.  3a  shows  the  system  model  for  acquiring  an 
undersampled  image  in  a  one-dimensional  case.  Let 
f  (x)  be  the  ideal  target  signature  that  is  interrogated 

by  the  sensor.  The  measured  target  signature  g'(x) 
by  the  sensor  is  modeled  as  the  output  of  a  Linear 


Shift  Invariant  (LSI)  system  whose  impulse  response 
is  the  sensor’s  point  spread  function  (PSF),  h(x)  ; 
this  is  also  known  as  the  sensor  blurring  function. 
The  relationship  between  the  measured  target 
signature  and  the  original  target  signature  in  the 

spatial  frequency  kx  domain  is: 

G(kx)  =  F(kx)H(kx ) 

where  G(kx),  F(kx)  ,  and  H{kx)  are  the  Fourier 
transforms  of  g(x) ,  /(x) ,  and  h(x) ,  respectively. 

Fig.  3b  shows  the  factors  that  dictate  the 
bandwidth  of  the  measured  target  signature.  The 
bandwidth  of  the  sensor’s  PSF  is  fixed  and  is 
determined  by  the  support  of  the  transfer 
function//^).  The  bandwidth  of  the  ideal  target 

signature  depends  on,  e.g.,  the  range  of  the  target  in 
FLIR,  radar,  sonar,  visible  light,  etc.  For  a  target  at 
the  short  range,  more  details  of  the  target  are 
observable;  in  this  case,  the  ideal  target  signature 
bandwidth  is  relatively  large.  For  a  target  at  the  long 
range,  the  ideal  target  signature  bandwidth  is  smaller. 
Two  curves  in  Fig.  3b  illustrate  the  bandwidths  of  the 
sensor  and  the  ideal  target  signature.  The  wider  one 
could  be  the  bandwidth  of  the  target  or  the  sensor,  or 
vise  versa.  The  output  bandwidth  of  the  measured 
target  signature  is  the  minimum  bandwidth  of  the 
ideal  target  signature  and  the  sensor,  that  is, 

where  Bo,  Bt,  and  Bs  are  the  bandwidths  of 

the  output  target  signature,  the  ideal  target  signature 
and  the  sensor  PSF,  respectively.  The  proposed 
super-resolution  image  reconstruction  algorithm 
cannot  produce  an  image  whose  bandwidth 

exceeds  Bo . 

One  of  the  issues  in  super-resolution  image 
reconstruction  is  the  number  of  the  undersampled 
images  that  are  required  to  recover  the  alias-free 
image.  Let  the  sampling  space  of  the  original 

undersampled  images  be  Ax  .  From  an  information- 

theory  point  of  view,  the  sampling  space  of  the 
output  high-resolution  image  that  is  reconstructed 
from  k  frames  (with  distinct  sub-pixel  shifts)  is 

^  in  one-dimensional  imaging  systems,  and 
k 

(  ?  )  in  two-dimensional  imaging  systems. 

4k  ’  4k 

Theoretically,  more  frames  should  provide  more 
information  and,  thus,  larger  bandwidth  and/or  higher 
resolution.  However,  the  super-resolution  image 
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quality  does  not  improve  when  the  number  of 
processed  undersampled  images  exceeds  a  certain 
value.  This  is  due  to  the  fact  discussed  above  that  the 
bandwidth  recovery  is  limited  by  the  minimum  of  the 
bandwidths  of  the  sensor  or  ideal  target  signature. 
Resolution  is  ultimately  limited  by  diffraction  and 
sensitivity  is  ultimately  limited  by  eye  noise. 

3.2.  Implementation  of  Error-Energy  Reduction 
Reconstruction  Algorithm 

Let  the  bandwidth  of  the  input  undersampled 
(aliased)  images  be  Bt .  The  bandwidth  of  the  alias- 

free  (high-resolution)  output  image  is  denoted  as  Bo  . 

In  order  to  recover  the  desired  (alias-free)  bandwidth, 
it  is  important  to  select  a  processing  array  with  a 
sample  spacing  smaller  than  the  sample  spacing  of 
the  desired  high-resolution  image.  In  this  case,  the 
2D  FFT  of  the  processing  array  yields  a  spectral 

coverage,  Bp  ,  that  is  larger  than  the  bandwidth  of  the 

output  image. 

Fig.  4  shows  a  diagram  of  the  error-energy 
reduction  algorithm.  In  the  first  step,  the  processing 
array  is  initialized  by  populating  the  grids  using  the 
input  images  according  to  the  estimates  of  the  sub¬ 
pixel  shifts.  Then,  we  form  the  2-D  Fourier  transform 
of  this  processing  array.  Its  spectrum  has  a  wider 
bandwidth  than  the  true  (desired)  output  bandwidth. 
Therefore,  the  spatial  frequency  domain  constraint  is 
applied,  that  is,  replacing  zeros  outside  the  desired 
bandwidth.  The  next  step  is  to  perform  the  inverse  2- 
D  Fourier  transform  of  the  resultant  array. 

The  resultant  inverse  array  is  not  a  true  image 
because  the  image  values  at  the  known  grid  locations 
are  not  equal  to  the  original  image  values.  Then  the 
spatial  domain  constraint  is  applied,  that  is,  replacing 
those  image  values  at  the  known  grid  locations  with 
the  known  original  image  values.  The  procedure 
continues  until  the  n  th  iteration. 

The  use  of  the  available  information  in  the 
spatial  and  spatial  frequency  domains  results  in  a 
reduction  of  the  energy  of  the  error  at  each  iteration 
step.  At  the  odd  steps  of  the  iteration,  the  error  is 
defined  by 

e2„+i  =  z  [/?2„+i  (*>  y)  -  p(x,  y)] 2 

(x,yUXp,Yp] 

where  p(x,y)  is  the  high-resolution  alias-free  image 
on  the  processing  array  (grid),  p2n+i(x’>y)  *s  the 


reconstructed  high-resolution  image  at  the  iteration 
2n  +  l. 

In  this  paper,  the  condition  of  the  error-energy 
reduction  is  examined  by  defining  the  following 
ratio: 


SNR2n+l=  101og10( 


(x,y)c[Xp,Yp] 


<x.frtxp,r„i 


-)  (1) 


lfSNR2n+l  <  SNR,mx  (where  SNRmJs  a  pre- 

assigned  threshold  value),  and  the  maximum  iteration 
is  not  reached,  the  iteration  continues. 
lfSNR2n+l  >  SNRmax ,  i.e.,  the  stopping  criterion  is 

satisfied,  the  iteration  is  terminated.  Before  the  final 
super-resolution  image  is  generated,  the  bandwidth  of 
the  output  image  is  reduced  from  Bp  to  Bo  using  the 

Fourier  windowing  method  (Young,  2004).  Then  the 
final  super-resolution  image  with  the  desired  alias- 
free  bandwidth  is  saved  for  the  output. 


4.  RESULTS 

In  this  section,  we  provide  several  examples  to 
demonstrate  the  merits  of  the  proposed  super¬ 
resolution  image  reconstruction  algorithm.  The  first 
example  is  to  use  the  simulated  low  resolution 
undersampled  images  from  a  high  resolution  FLIR 
image.  Because  the  sub-pixel  shift  information  for 
these  simulated  low  resolution  images  is  pre¬ 
determined,  we  can  test  the  accuracy  of  the  proposed 
sub-pixel  shift  estimate  algorithm  and  the  error- 
energy  reduction  reconstruction  algorithm.  Then  real 
world  images  of  FLIR  sensors  are  utilized  to  test  the 
proposed  algorithm  and  demonstrate  the  merits  of  the 
proposed  algorithm. 

4.1  Simulated  Low  Resolution  FLIR  Images 

The  original  FLIR  tank  image  is  shown  in  Fig. 
5a.  The  size  of  this  image  is  40  pixels  in  the  vertical 
dimension  and  76  pixels  in  the  horizontal  dimension. 
First,  we  up-sample  this  image  using  the  Fourier 
windowing  method  (Young,  2004)  by  a  factor  of  4  to 
obtain  a  simulated  high  resolution  image  with  a  size 
of  160x304  as  shown  in  Fig.  5b.  Then  the  low 
resolution  images  are  generated  from  this  simulated 
high  resolution  image  by  sub-sampling;  the  starting 
sample  point  (shift)  is  randomly  generated  for  each 
low  resolution  image. 

The  size  of  the  low  resolution  image  is  20x38. 
The  resolution  factor  between  the  simulated  high 
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resolution  image  and  the  low  resolution  images  is  8. 
The  low  resolution  images  are  formed  by  sub¬ 
sampling  every  8  pixels  in  both  dimensions  from  the 
simulated  high  resolution  image.  The  first  samples  of 
the  4  low  resolution  images  in  the  high  resolution 
upsampled  image  are  at  (1,  1),  (3,  2),  (2,  5),  and  (7, 
3),  that  represent  the  sub-pixel  shift  locations.  The 
low  resolution  images  are  sub-sampled  from  the 
simulated  high  resolution  image  based  on  the  pre¬ 
described  sub-pixel  location  points.  For  example,  the 
simulated  low  resolution  image  No.  2  is  formed  by 
sub-sampling  the  simulated  high  resolution  image 
starting  at  the  3rd  sub-pixel  position  in  the  first 
dimension  and  2nd  sub-pixel  position  in  the  second 
dimension.  Four  simulated  low  resolution  images  are 
shown  in  Fig.  5c-f. 

After  applying  the  sub-pixel  shift  estimation 
algorithm  to  these  4  simulated  low  resolution  images, 
the  sub-pixel  shift  values  are  obtained.  The  estimated 
sub-pixel  shift  values  are  all  correct  according  to  the 
actual  shifts. 

The  reconstructed  image  from  the  error-energy 
reduction  algorithm  is  shown  in  Fig.  5j.  For 
comparison  purposes,  we  also  applied  a  standard 
interpolation  algorithm  based  on  the  gradient 
estimated  sub-pixel  shifts  among  these  4  low- 
resolution  images.  The  resultant  image  is  shown  in 
Fig.  5i.  This  image  shows  artifacts  that  could  be  due 
to  the  fact  that  the  sub-pixel  shifts  are  not  accurately 
estimated  and  the  number  of  the  input  low  resolution 
images  is  small  (It  is  stated  in  the  literature  that  these 
algorithms  require  a  large  number  of  snap  shots 
(Schuler  et  al.,  2002).  The  ratio  of  signal  to  error 
(defined  in  the  equation  (1))  between  the  high- 
resolution  output  image  in  Fig.  5i  and  the  original 
image  in  Fig.  5a  is  0.56dB.  This  indicates  that  the 
error  between  the  high-resolution  output  image  and 
the  original  image  is  large.  Fig.  5g-h  show  one  of  the 
low-resolution  images  and  its  bilinear  interpolated 
image.  The  ratio  of  signal  to  error  between  the 
bilinear  interpolated  image  in  Fig.  5h  and  the  original 
image  in  Fig.  5a  is  0.71dB.  This  indicates  that  the 
error  between  the  interpolated  low  resolution  image 
and  the  original  image  is  also  large.  The  ratio  of 
signal  to  error  between  the  super-resolved  image  in 
Fig.  5j  and  the  original  image  in  Fig.  5a  is  40.1dB. 
This  indicates  that  the  image  formed  via  the  proposed 
algorithm  is  a  good  estimate  of  the  original  image. 
By  observing  Fig.  5j  and  Fig.  5h,  we  can  see  that  the 
super-resolved  image  shows  a  significant 
improvement  by  exhibiting  the  detailed  information, 
especially  around  the  road  wheel  area,  of  the  tank. 


4.2  FLIR  Images 

A  sequence  of  FLIR  images  of  a  foliage  area  is 
acquired  using  a  FLIR  sensor,  Indigo  Systems  Merlin 
LWIR  uncooled  microbolometer  thermographic 
camera.  The  sub-pixel  shifts  among  the  frames  are 
due  to  the  natural  jitter  of  the  camera.  Sixteen  low 
resolution  frames  are  used  to  reconstruct  the  high- 
resolution  (de-aliased)  image  with  the  resolution 
improvement  factor  of  4,  as  shown  in  Fig.  6b.  For 
comparison,  one  of  the  bilinear  interpolated  low 
resolution  images  is  shown  in  Fig.  6a.  The 
reconstructed  image  in  Fig.  6b  shows  a  significant 
improvement  by  revealing  high  frequency 
information  on  the  tree  branches. 

Next  we  consider  a  sequence  of  FLIR  images  of 
a  pickup  truck  that  is  acquired  using  the  same  FLIR 
sensor  that  was  mentioned  above.  Similarly,  the 
natural  jittering  camera  produces  the  sub-pixel  shifts 
among  the  frames.  Fig.  7a  shows  a  bilinear 
interpolated  image  from  one  of  the  low  resolution 
images.  Fig.  7b  shows  the  high-resolution  pickup 
truck  image  that  is  reconstructed  using  16  low 
resolution  images.  This  reconstructed  image  again 
illustrates  a  significant  improvement  by  exhibiting 
the  detailed  information  on  the  cabin  and  window 
area  of  the  truck  which  contains  rich  high  frequency 
information. 

5.  CONCLUSION 

This  paper  proposes  an  iterative  error-energy 
reduction  algorithm  to  reconstruct  the  high-resolution 
(alias-free)  output  image  that  utilizes  a  correlation 
method  to  estimate  the  sub-pixel  shifts  among  a 
sequence  of  low  resolution  aliased  imagery.  The 
proposed  super-resolution  image  reconstruction 
algorithm  provides  a  possibility  to  produce  high- 
resolution  images  by  using  low  resolution  images 
from  existing  low-cost  imaging  devices.  Our 
numerical  study  has  indicated  that  our  super¬ 
resolution  image  reconstruction  algorithm  can 
produce  high-resolution  images  using  4  frames 
(doubling  the  original  sampling  rate)  to  16  frames 
(quadrupling  the  original  sampling  rate).  Whereas 
most  other  super-resolution  image  reconstruction 
methods  require  over  100  frames  to  achieve  a  similar 
performance  (Schuler,  2002).  In  addition  to  the 
accuracy  and  robustness  (stability)  of  the  solution  of 
our  algorithm,  the  ability  to  use  a  relatively  small 
number  of  frames  that  are  acquired  at  sub-pixel  shifts 
(instead  of  a  continuous  camera  motion  that  is  also 
required  by  many  of  the  existing  algorithms)  brings  a 
flexibility  that  is  crucial  in  practical  scenarios 
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(operational  systems),  e.g.,  the  Army’s  weapon 
system  that  requires  low  memory  and  fast  processing. 
A  human  observer  study  was  performed  (Krapels  et. 
ah,  2004)  that  demonstrated  the  performance 
improvement  of  the  proposed  super-resolution  image 
reconstruction  algorithm  for  undersampled  FLIR 
imagers. 
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Input  low 
resolution  image 


Output  high 
resolution  image 
(sequence  or 
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Fig.  1  Overview  of  the  super-resolution  image 
reconstruction  algorithm. 
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Fig.  3a  Image  acquisition  model. 


Fig.3b  Factors  that  indicate  the  bandwidth  of  the 
measured  target  signature. 
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Fig. 2  An  example  of  sub-pixel  shift  estimation 
from  16  frames.  The  numbers  on  the  figure  represents 
the  names  of  frames.  The  sub-pixel  shifts  of  each 
frame  with  respect  to  a  reference  frame  are  illustrated 
at  their  corresponding  sub-pixel  locations. 
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Fig.  4  Error-energy  reduction  algorithm. 
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(a) 
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Fig.  6  FLIR  images  of  a  foliage  area,  (a)  Bilinear 
interpolated  image,  (b)  Super-resolved  image  by  the 
proposed  algorithm. 


Fig.  5  Simulated  low  resolution  FLIR  images,  (a) 
Original  FLIR  tank  image,  (b)  Simulated  high 
resolution  image,  (c)  -  (f)  Four  simulated  low 
resolution  images,  (g)  One  of  the  simulated  low 
resolution  images,  (h)  One  of  the  bilinear  interpolated 
low  resolution  images,  (i)  High-resolution  output 
image  by  the  standard  interpolation  method  based  on 
the  gradient  estimated  sub-pixel  shifts,  (j)  Super- 
resolved  image  by  the  proposed  algorithm. 


Fig.  7  FLIR  images  of  a  pickup  truck,  (a) 
Bilinear  interpolated  image,  (b)  Super-resolved 
image  by  the  proposed  algorithm. 
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